VLM Finetuning support #411
Open
+388
−88
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikita-smetanin
commented
Dec 22, 2025
nikita-smetanin
commented
- Add support for Multimodal datasets in OpenAI-like format
- Add support for Vision-Language model training with optional Vision encoder finetuning
sbassam
approved these changes
Dec 23, 2025
| elif messages_are_multimodal != is_multimodal: | ||
| # Due to the format limitation, we cannot mix multimodal and text only messages in the same sample. | ||
| raise InvalidFileFormatError( | ||
| "Messages in the conversation must be either all in multimodal or all intext only format.", |
Contributor
There was a problem hiding this comment.
nit: typo: ...or all in text-only
Comment on lines
+273
to
+274
| message: The message to check. | ||
| idx: Line number in the file. |
connermanuel
approved these changes
Dec 23, 2025
Contributor
There was a problem hiding this comment.
certainly dont mind this change but i wonder how it got in
|
|
||
| if model_limits.supports_vision: | ||
| # Don't show price estimation for multimodal models yet | ||
| confirm = True |
Contributor
There was a problem hiding this comment.
sorry, i don't have context here, why does this prevent showing the price estimation?
|
|
||
| if model_limits.supports_vision: | ||
| multimodal_params = FinetuneMultimodalParams(train_vision=train_vision) | ||
| elif train_vision: |
Contributor
There was a problem hiding this comment.
supernit: i prefer
elif not model_limits.supports_vision and train_vision
here. it's logically the same, but the condition is clearer
| line_number=idx + 1, | ||
| error_source="key_value", | ||
| ) | ||
| if not isinstance(message[column], str): |
Contributor
There was a problem hiding this comment.
perhaps you can check isinstance(message[column], MessageContent) instead?
| def _check_message_role( | ||
| message: Dict[str, str | bool], previous_role: str | None, idx: int | ||
| ) -> str | bool: | ||
| message: Dict[str, str | int | MessageContent], previous_role: str | None, idx: int |
Contributor
There was a problem hiding this comment.
when is the message an int?
|
|
||
|
|
||
| def _check_message_content( | ||
| message_content: str | int | MessageContent, role: str, idx: int |
Contributor
There was a problem hiding this comment.
when is the message an int?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.